

<!DOCTYPE html>
<html class="writer-html5" lang="zh" >
<head>
  <meta charset="utf-8">
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  
  <title>Scrapy shell &mdash; Scrapy 2.3.0 文档</title>
  

  
  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster.custom.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster.bundle.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-shadow.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-punk.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-noir.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-light.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-borderless.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/micromodal.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/sphinx_rtd_theme.css" type="text/css" />

  
  
  
  

  
  <!--[if lt IE 9]>
    <script src="../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
        <script src="../_static/jquery.js"></script>
        <script src="../_static/underscore.js"></script>
        <script src="../_static/doctools.js"></script>
        <script src="../_static/language_data.js"></script>
        <script src="../_static/js/hoverxref.js"></script>
        <script src="../_static/js/tooltipster.bundle.min.js"></script>
        <script src="../_static/js/micromodal.min.js"></script>
    
    <script type="text/javascript" src="../_static/js/theme.js"></script>

    
    <link rel="index" title="索引" href="../genindex.html" />
    <link rel="search" title="搜索" href="../search.html" />
    <link rel="next" title="项目管道" href="item-pipeline.html" />
    <link rel="prev" title="项目加载器" href="loaders.html" /> 
</head>

<body class="wy-body-for-nav">

   
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search" >
          

          
            <a href="../index.html" class="icon icon-home" alt="Documentation Home"> Scrapy
          

          
          </a>

          
            
            
              <div class="version">
                2.3
              </div>
            
          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <p class="caption"><span class="caption-text">第一步</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../intro/overview.html">Scrapy一目了然</a></li>
<li class="toctree-l1"><a class="reference internal" href="../intro/install.html">安装指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../intro/tutorial.html">Scrapy 教程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../intro/examples.html">实例</a></li>
</ul>
<p class="caption"><span class="caption-text">基本概念</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="commands.html">命令行工具</a></li>
<li class="toctree-l1"><a class="reference internal" href="spiders.html">蜘蛛</a></li>
<li class="toctree-l1"><a class="reference internal" href="selectors.html">选择器</a></li>
<li class="toctree-l1"><a class="reference internal" href="items.html">项目</a></li>
<li class="toctree-l1"><a class="reference internal" href="loaders.html">项目加载器</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Scrapy shell</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#configuring-the-shell">配置shell</a></li>
<li class="toctree-l2"><a class="reference internal" href="#launch-the-shell">启动外壳</a></li>
<li class="toctree-l2"><a class="reference internal" href="#using-the-shell">使用外壳</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#available-shortcuts">可用快捷方式</a></li>
<li class="toctree-l3"><a class="reference internal" href="#available-scrapy-objects">可用的 Scrapy 对象</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#example-of-shell-session">Shell会话示例</a></li>
<li class="toctree-l2"><a class="reference internal" href="#invoking-the-shell-from-spiders-to-inspect-responses">从spiders调用shell来检查响应</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="item-pipeline.html">项目管道</a></li>
<li class="toctree-l1"><a class="reference internal" href="feed-exports.html">Feed 导出</a></li>
<li class="toctree-l1"><a class="reference internal" href="request-response.html">请求和响应</a></li>
<li class="toctree-l1"><a class="reference internal" href="link-extractors.html">链接提取器</a></li>
<li class="toctree-l1"><a class="reference internal" href="settings.html">设置</a></li>
<li class="toctree-l1"><a class="reference internal" href="exceptions.html">例外情况</a></li>
</ul>
<p class="caption"><span class="caption-text">内置服务</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="logging.html">登录</a></li>
<li class="toctree-l1"><a class="reference internal" href="stats.html">统计数据集合</a></li>
<li class="toctree-l1"><a class="reference internal" href="email.html">发送电子邮件</a></li>
<li class="toctree-l1"><a class="reference internal" href="telnetconsole.html">远程登录控制台</a></li>
<li class="toctree-l1"><a class="reference internal" href="webservice.html">Web服务</a></li>
</ul>
<p class="caption"><span class="caption-text">解决具体问题</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">常见问题</a></li>
<li class="toctree-l1"><a class="reference internal" href="debug.html">调试spiders</a></li>
<li class="toctree-l1"><a class="reference internal" href="contracts.html">蜘蛛合约</a></li>
<li class="toctree-l1"><a class="reference internal" href="practices.html">常用做法</a></li>
<li class="toctree-l1"><a class="reference internal" href="broad-crawls.html">宽爬行</a></li>
<li class="toctree-l1"><a class="reference internal" href="developer-tools.html">使用浏览器的开发人员工具进行抓取</a></li>
<li class="toctree-l1"><a class="reference internal" href="dynamic-content.html">选择动态加载的内容</a></li>
<li class="toctree-l1"><a class="reference internal" href="leaks.html">调试内存泄漏</a></li>
<li class="toctree-l1"><a class="reference internal" href="media-pipeline.html">下载和处理文件和图像</a></li>
<li class="toctree-l1"><a class="reference internal" href="deploy.html">部署蜘蛛</a></li>
<li class="toctree-l1"><a class="reference internal" href="autothrottle.html">AutoThrottle 扩展</a></li>
<li class="toctree-l1"><a class="reference internal" href="benchmarking.html">标杆管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="jobs.html">作业：暂停和恢复爬行</a></li>
<li class="toctree-l1"><a class="reference internal" href="coroutines.html">协同程序</a></li>
<li class="toctree-l1"><a class="reference internal" href="asyncio.html">asyncio</a></li>
</ul>
<p class="caption"><span class="caption-text">扩展Scrapy</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="architecture.html">体系结构概述</a></li>
<li class="toctree-l1"><a class="reference internal" href="downloader-middleware.html">下载器中间件</a></li>
<li class="toctree-l1"><a class="reference internal" href="spider-middleware.html">蜘蛛中间件</a></li>
<li class="toctree-l1"><a class="reference internal" href="extensions.html">扩展</a></li>
<li class="toctree-l1"><a class="reference internal" href="api.html">核心API</a></li>
<li class="toctree-l1"><a class="reference internal" href="signals.html">信号</a></li>
<li class="toctree-l1"><a class="reference internal" href="exporters.html">条目导出器</a></li>
</ul>
<p class="caption"><span class="caption-text">其余所有</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../news.html">发行说明</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">为 Scrapy 贡献</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versioning.html">版本控制和API稳定性</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">Scrapy</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
        
      <li>Scrapy shell</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
            
        
      </li>
    
  </ul>

  
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
  <div class="section" id="scrapy-shell">
<span id="topics-shell"></span><h1>Scrapy shell<a class="headerlink" href="#scrapy-shell" title="永久链接至标题">¶</a></h1>
<p>scrappyshell是一个交互式shell，您可以在其中快速调试 scrape 代码，而不必运行spider。它本来是用来测试数据提取代码的，但实际上您可以使用它来测试任何类型的代码，因为它也是一个常规的Python外壳。</p>
<p>shell用于测试xpath或css表达式，并查看它们是如何工作的，以及它们从您试图抓取的网页中提取的数据。它允许您在编写spider时交互地测试表达式，而不必运行spider来测试每个更改。</p><script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>
<ins class="adsbygoogle"
     style="display:block; text-align:center;"
     data-ad-layout="in-article"
     data-ad-format="fluid"
     data-ad-client="ca-pub-1466963416408457"
     data-ad-slot="8850786025"></ins>
<script>
     (adsbygoogle = window.adsbygoogle || []).push({});
</script>
<p>一旦你熟悉了 Scrapy Shell，你就会发现它是开发和调试蜘蛛的宝贵工具。</p>
<div class="section" id="configuring-the-shell">
<h2>配置shell<a class="headerlink" href="#configuring-the-shell" title="永久链接至标题">¶</a></h2>
<p>如果你有 <a class="reference external" href="https://ipython.org/">IPython</a> 安装后，scrapy shell将使用它（而不是标准的python控制台）。这个 <a class="reference external" href="https://ipython.org/">IPython</a> 控制台功能更强大，提供智能自动完成和彩色输出等功能。</p>
<p>我们强烈建议您安装 <a class="reference external" href="https://ipython.org/">IPython</a> ，特别是在使用Unix系统时（其中 <a class="reference external" href="https://ipython.org/">IPython</a> 擅长）。见 <a class="reference external" href="https://ipython.org/install.html">IPython installation guide</a> 更多信息。</p>
<p>Scrapy还支持 <a class="reference external" href="https://bpython-interpreter.org/">bpython</a> ，并将尝试在 <a class="reference external" href="https://ipython.org/">IPython</a> 不可用。</p>
<p>通过Scrapy的设置，你可以配置它使用 <code class="docutils literal notranslate"><span class="pre">ipython</span></code> ， <code class="docutils literal notranslate"><span class="pre">bpython</span></code> 或标准 <code class="docutils literal notranslate"><span class="pre">python</span></code> 外壳，无论安装了什么。这是通过设置 <code class="docutils literal notranslate"><span class="pre">SCRAPY_PYTHON_SHELL</span></code> 环境变量；或通过在 <a class="reference internal" href="commands.html#topics-config-settings"><span class="std std-ref">scrapy.cfg</span></a> ：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[</span><span class="n">settings</span><span class="p">]</span>
<span class="n">shell</span> <span class="o">=</span> <span class="n">bpython</span>
</pre></div>
</div>
</div>
<div class="section" id="launch-the-shell">
<h2>启动外壳<a class="headerlink" href="#launch-the-shell" title="永久链接至标题">¶</a></h2>
<p>要启动碎屑壳，可以使用 <a class="reference internal" href="commands.html#std-command-shell"><code class="xref std std-command docutils literal notranslate"><span class="pre">shell</span></code></a> 命令如下：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">scrapy</span> <span class="n">shell</span> <span class="o">&lt;</span><span class="n">url</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>何处 <code class="docutils literal notranslate"><span class="pre">&lt;url&gt;</span></code> 是要擦除的URL。</p>
<p><a class="reference internal" href="commands.html#std-command-shell"><code class="xref std std-command docutils literal notranslate"><span class="pre">shell</span></code></a> 也适用于本地文件。如果你想玩一个网页的本地副本，这很方便。 <a class="reference internal" href="commands.html#std-command-shell"><code class="xref std std-command docutils literal notranslate"><span class="pre">shell</span></code></a> 了解本地文件的以下语法：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># UNIX-style</span>
<span class="n">scrapy</span> <span class="n">shell</span> <span class="o">./</span><span class="n">path</span><span class="o">/</span><span class="n">to</span><span class="o">/</span><span class="n">file</span><span class="o">.</span><span class="n">html</span>
<span class="n">scrapy</span> <span class="n">shell</span> <span class="o">../</span><span class="n">other</span><span class="o">/</span><span class="n">path</span><span class="o">/</span><span class="n">to</span><span class="o">/</span><span class="n">file</span><span class="o">.</span><span class="n">html</span>
<span class="n">scrapy</span> <span class="n">shell</span> <span class="o">/</span><span class="n">absolute</span><span class="o">/</span><span class="n">path</span><span class="o">/</span><span class="n">to</span><span class="o">/</span><span class="n">file</span><span class="o">.</span><span class="n">html</span>

<span class="c1"># File URI</span>
<span class="n">scrapy</span> <span class="n">shell</span> <span class="n">file</span><span class="p">:</span><span class="o">///</span><span class="n">absolute</span><span class="o">/</span><span class="n">path</span><span class="o">/</span><span class="n">to</span><span class="o">/</span><span class="n">file</span><span class="o">.</span><span class="n">html</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>使用相对文件路径时，请显式并用 <code class="docutils literal notranslate"><span class="pre">./</span></code> （或） <code class="docutils literal notranslate"><span class="pre">../</span></code> 相关时）。 <code class="docutils literal notranslate"><span class="pre">scrapy</span> <span class="pre">shell</span> <span class="pre">index.html</span></code> 不会像人们预期的那样工作（这是设计上的，而不是错误）。</p>
<p>因为 <a class="reference internal" href="commands.html#std-command-shell"><code class="xref std std-command docutils literal notranslate"><span class="pre">shell</span></code></a> 喜欢HTTP URL而不是文件URI，以及 <code class="docutils literal notranslate"><span class="pre">index.html</span></code> 在句法上类似于 <code class="docutils literal notranslate"><span class="pre">example.com</span></code> ， <a class="reference internal" href="commands.html#std-command-shell"><code class="xref std std-command docutils literal notranslate"><span class="pre">shell</span></code></a> 会治疗 <code class="docutils literal notranslate"><span class="pre">index.html</span></code> 作为域名并触发DNS查找错误：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ scrapy shell index.html
[ ... scrapy shell starts ... ]
[ ... traceback ... ]
twisted.internet.error.DNSLookupError: DNS lookup failed:
address &#39;index.html&#39; not found: [Errno -5] No address associated with hostname.
</pre></div>
</div>
<p><a class="reference internal" href="commands.html#std-command-shell"><code class="xref std std-command docutils literal notranslate"><span class="pre">shell</span></code></a> 如果文件调用了 <code class="docutils literal notranslate"><span class="pre">index.html</span></code> 存在于当前目录中。同样，要明确。</p>
</div>
</div>
<div class="section" id="using-the-shell">
<h2>使用外壳<a class="headerlink" href="#using-the-shell" title="永久链接至标题">¶</a></h2>
<p>scrappyshell只是一个普通的python控制台（或者 <a class="reference external" href="https://ipython.org/">IPython</a> 控制台，如果你有它的话），它提供一些额外的快捷功能，以方便。</p>
<div class="section" id="available-shortcuts">
<h3>可用快捷方式<a class="headerlink" href="#available-shortcuts" title="永久链接至标题">¶</a></h3>
<blockquote>
<div><ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">shelp()</span></code> -打印有关可用对象和快捷方式列表的帮助</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">fetch(url[,</span> <span class="pre">redirect=True])</span></code> - fetch a new response from the given
URL and update all related objects accordingly. You can optionaly ask for
HTTP 3xx redirections to not be followed by passing <code class="docutils literal notranslate"><span class="pre">redirect=False</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">fetch(request)</span></code> -从给定的请求中获取新的响应，并相应地更新所有相关对象。</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">view(response)</span></code> -在本地Web浏览器中打开给定的响应以进行检查。这将增加一个 <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/base">&lt;base&gt; tag</a> 到响应主体，以便外部链接（如图像和样式表）正确显示。但是请注意，这将在您的计算机中创建一个临时文件，该文件不会自动删除。</p></li>
</ul>
</div></blockquote>
</div>
<div class="section" id="available-scrapy-objects">
<h3>可用的 Scrapy 对象<a class="headerlink" href="#available-scrapy-objects" title="永久链接至标题">¶</a></h3>
<p>Scrapy Shell自动从下载的页面创建一些方便的对象，例如 <a class="reference internal" href="request-response.html#scrapy.http.Response" title="scrapy.http.Response"><code class="xref py py-class docutils literal notranslate"><span class="pre">Response</span></code></a> 对象与 <a class="reference internal" href="selectors.html#scrapy.selector.Selector" title="scrapy.selector.Selector"><code class="xref py py-class docutils literal notranslate"><span class="pre">Selector</span></code></a> 对象（用于HTML和XML内容）。</p>
<p>这些对象是：</p>
<blockquote>
<div><ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">crawler</span></code> -电流 <a class="reference internal" href="api.html#scrapy.crawler.Crawler" title="scrapy.crawler.Crawler"><code class="xref py py-class docutils literal notranslate"><span class="pre">Crawler</span></code></a> 对象。</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">spider</span></code> -已知用于处理URL的蜘蛛，或 <a class="reference internal" href="spiders.html#scrapy.spiders.Spider" title="scrapy.spiders.Spider"><code class="xref py py-class docutils literal notranslate"><span class="pre">Spider</span></code></a> 如果没有为当前URL找到spider，则为。</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">request</span></code> -A <a class="reference internal" href="request-response.html#scrapy.http.Request" title="scrapy.http.Request"><code class="xref py py-class docutils literal notranslate"><span class="pre">Request</span></code></a> 上次提取的页的对象。您可以使用修改此请求 <a class="reference internal" href="request-response.html#scrapy.http.Request.replace" title="scrapy.http.Request.replace"><code class="xref py py-meth docutils literal notranslate"><span class="pre">replace()</span></code></a> 或者使用 <code class="docutils literal notranslate"><span class="pre">fetch</span></code> 捷径。</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">response</span></code> -A <a class="reference internal" href="request-response.html#scrapy.http.Response" title="scrapy.http.Response"><code class="xref py py-class docutils literal notranslate"><span class="pre">Response</span></code></a> 包含上次提取的页的对象</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">settings</span></code> - the current <a class="reference internal" href="settings.html#topics-settings"><span class="std std-ref">Scrapy settings</span></a></p></li>
</ul>
</div></blockquote>
</div>
</div>
<div class="section" id="example-of-shell-session">
<h2>Shell会话示例<a class="headerlink" href="#example-of-shell-session" title="永久链接至标题">¶</a></h2>
<p>下面是一个典型的shell会话的示例，我们从https://scray.org页面，然后继续刮https://old.reddit.com/第页。最后，我们修改（Reddit）请求方法，以发布并重新获取它，得到一个错误。我们通过键入Ctrl-D（在Unix系统中）或在Windows中键入Ctrl-Z来结束会话。</p>
<p>请记住，在这里提取的数据在您尝试时可能不相同，因为这些页面不是静态的，在您测试时可能已经更改了。这个例子的唯一目的是让您熟悉下脚料外壳的工作原理。</p>
<p>首先，我们发射炮弹：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">scrapy</span> <span class="n">shell</span> <span class="s1">&#39;https://scrapy.org&#39;</span> <span class="o">--</span><span class="n">nolog</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>当从命令行运行Scrapy shell时，记住总是用引号括住url，否则url包含参数（即 <code class="docutils literal notranslate"><span class="pre">&amp;</span></code> 字符）不起作用。</p>
<p>在Windows上，使用双引号：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">scrapy</span> <span class="n">shell</span> <span class="s2">&quot;https://scrapy.org&quot;</span> <span class="o">--</span><span class="n">nolog</span>
</pre></div>
</div>
</div>
<p>然后，shell获取URL（使用scrapy下载器）并打印可用对象和有用快捷方式的列表（您会注意到这些行都以 <code class="docutils literal notranslate"><span class="pre">[s]</span></code> 前缀）：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[</span><span class="n">s</span><span class="p">]</span> <span class="n">Available</span> <span class="n">Scrapy</span> <span class="n">objects</span><span class="p">:</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">scrapy</span>     <span class="n">scrapy</span> <span class="n">module</span> <span class="p">(</span><span class="n">contains</span> <span class="n">scrapy</span><span class="o">.</span><span class="n">Request</span><span class="p">,</span> <span class="n">scrapy</span><span class="o">.</span><span class="n">Selector</span><span class="p">,</span> <span class="n">etc</span><span class="p">)</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">crawler</span>    <span class="o">&lt;</span><span class="n">scrapy</span><span class="o">.</span><span class="n">crawler</span><span class="o">.</span><span class="n">Crawler</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x7f07395dd690</span><span class="o">&gt;</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">item</span>       <span class="p">{}</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">request</span>    <span class="o">&lt;</span><span class="n">GET</span> <span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">scrapy</span><span class="o">.</span><span class="n">org</span><span class="o">&gt;</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">response</span>   <span class="o">&lt;</span><span class="mi">200</span> <span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">scrapy</span><span class="o">.</span><span class="n">org</span><span class="o">/&gt;</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">settings</span>   <span class="o">&lt;</span><span class="n">scrapy</span><span class="o">.</span><span class="n">settings</span><span class="o">.</span><span class="n">Settings</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x7f07395dd710</span><span class="o">&gt;</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">spider</span>     <span class="o">&lt;</span><span class="n">DefaultSpider</span> <span class="s1">&#39;default&#39;</span> <span class="n">at</span> <span class="mh">0x7f0735891690</span><span class="o">&gt;</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span> <span class="n">Useful</span> <span class="n">shortcuts</span><span class="p">:</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">fetch</span><span class="p">(</span><span class="n">url</span><span class="p">[,</span> <span class="n">redirect</span><span class="o">=</span><span class="kc">True</span><span class="p">])</span> <span class="n">Fetch</span> <span class="n">URL</span> <span class="ow">and</span> <span class="n">update</span> <span class="n">local</span> <span class="n">objects</span> <span class="p">(</span><span class="n">by</span> <span class="n">default</span><span class="p">,</span> <span class="n">redirects</span> <span class="n">are</span> <span class="n">followed</span><span class="p">)</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">fetch</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>                  <span class="n">Fetch</span> <span class="n">a</span> <span class="n">scrapy</span><span class="o">.</span><span class="n">Request</span> <span class="ow">and</span> <span class="n">update</span> <span class="n">local</span> <span class="n">objects</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">shelp</span><span class="p">()</span>           <span class="n">Shell</span> <span class="n">help</span> <span class="p">(</span><span class="nb">print</span> <span class="n">this</span> <span class="n">help</span><span class="p">)</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">view</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>    <span class="n">View</span> <span class="n">response</span> <span class="ow">in</span> <span class="n">a</span> <span class="n">browser</span>

<span class="o">&gt;&gt;&gt;</span>
</pre></div>
</div>
<p>之后，我们可以开始玩对象：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//title/text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;Scrapy | A Fast and Powerful Scraping and Web Crawling Framework&#39;</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">fetch</span><span class="p">(</span><span class="s2">&quot;https://old.reddit.com/&quot;</span><span class="p">)</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//title/text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;reddit: the front page of the internet&#39;</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">request</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">method</span><span class="o">=</span><span class="s2">&quot;POST&quot;</span><span class="p">)</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">fetch</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">status</span>
<span class="go">404</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">pprint</span> <span class="kn">import</span> <span class="n">pprint</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">pprint</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">headers</span><span class="p">)</span>
<span class="go">{&#39;Accept-Ranges&#39;: [&#39;bytes&#39;],</span>
<span class="go"> &#39;Cache-Control&#39;: [&#39;max-age=0, must-revalidate&#39;],</span>
<span class="go"> &#39;Content-Type&#39;: [&#39;text/html; charset=UTF-8&#39;],</span>
<span class="go"> &#39;Date&#39;: [&#39;Thu, 08 Dec 2016 16:21:19 GMT&#39;],</span>
<span class="go"> &#39;Server&#39;: [&#39;snooserv&#39;],</span>
<span class="go"> &#39;Set-Cookie&#39;: [&#39;loid=KqNLou0V9SKMX4qb4n; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure&#39;,</span>
<span class="go">                &#39;loidcreated=2016-12-08T16%3A21%3A19.445Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure&#39;,</span>
<span class="go">                &#39;loid=vi0ZVe4NkxNWdlH7r7; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure&#39;,</span>
<span class="go">                &#39;loidcreated=2016-12-08T16%3A21%3A19.459Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure&#39;],</span>
<span class="go"> &#39;Vary&#39;: [&#39;accept-encoding&#39;],</span>
<span class="go"> &#39;Via&#39;: [&#39;1.1 varnish&#39;],</span>
<span class="go"> &#39;X-Cache&#39;: [&#39;MISS&#39;],</span>
<span class="go"> &#39;X-Cache-Hits&#39;: [&#39;0&#39;],</span>
<span class="go"> &#39;X-Content-Type-Options&#39;: [&#39;nosniff&#39;],</span>
<span class="go"> &#39;X-Frame-Options&#39;: [&#39;SAMEORIGIN&#39;],</span>
<span class="go"> &#39;X-Moose&#39;: [&#39;majestic&#39;],</span>
<span class="go"> &#39;X-Served-By&#39;: [&#39;cache-cdg8730-CDG&#39;],</span>
<span class="go"> &#39;X-Timer&#39;: [&#39;S1481214079.394283,VS0,VE159&#39;],</span>
<span class="go"> &#39;X-Ua-Compatible&#39;: [&#39;IE=edge&#39;],</span>
<span class="go"> &#39;X-Xss-Protection&#39;: [&#39;1; mode=block&#39;]}</span>
</pre></div>
</div>
</div>
<div class="section" id="invoking-the-shell-from-spiders-to-inspect-responses">
<span id="topics-shell-inspect-response"></span><h2>从spiders调用shell来检查响应<a class="headerlink" href="#invoking-the-shell-from-spiders-to-inspect-responses" title="永久链接至标题">¶</a></h2>
<p>有时，您希望检查在您的蜘蛛的某个点上正在处理的响应，如果只是检查您期望的响应是否到达那里的话。</p>
<p>这可以通过使用 <code class="docutils literal notranslate"><span class="pre">scrapy.shell.inspect_response</span></code> 功能。</p>
<p>下面是一个例子，说明如何从您的蜘蛛中命名它：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">scrapy</span>


<span class="k">class</span> <span class="nc">MySpider</span><span class="p">(</span><span class="n">scrapy</span><span class="o">.</span><span class="n">Spider</span><span class="p">):</span>
    <span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;myspider&quot;</span>
    <span class="n">start_urls</span> <span class="o">=</span> <span class="p">[</span>
        <span class="s2">&quot;http://example.com&quot;</span><span class="p">,</span>
        <span class="s2">&quot;http://example.org&quot;</span><span class="p">,</span>
        <span class="s2">&quot;http://example.net&quot;</span><span class="p">,</span>
    <span class="p">]</span>

    <span class="k">def</span> <span class="nf">parse</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">response</span><span class="p">):</span>
        <span class="c1"># We want to inspect one specific response.</span>
        <span class="k">if</span> <span class="s2">&quot;.org&quot;</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">url</span><span class="p">:</span>
            <span class="kn">from</span> <span class="nn">scrapy.shell</span> <span class="kn">import</span> <span class="n">inspect_response</span>
            <span class="n">inspect_response</span><span class="p">(</span><span class="n">response</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span>

        <span class="c1"># Rest of parsing code.</span>
</pre></div>
</div>
<p>当你运行蜘蛛时，你会得到类似的东西：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="mi">2014</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">23</span> <span class="mi">17</span><span class="p">:</span><span class="mi">48</span><span class="p">:</span><span class="mi">31</span><span class="o">-</span><span class="mi">0400</span> <span class="p">[</span><span class="n">scrapy</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">engine</span><span class="p">]</span> <span class="n">DEBUG</span><span class="p">:</span> <span class="n">Crawled</span> <span class="p">(</span><span class="mi">200</span><span class="p">)</span> <span class="o">&lt;</span><span class="n">GET</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">example</span><span class="o">.</span><span class="n">com</span><span class="o">&gt;</span> <span class="p">(</span><span class="n">referer</span><span class="p">:</span> <span class="kc">None</span><span class="p">)</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">23</span> <span class="mi">17</span><span class="p">:</span><span class="mi">48</span><span class="p">:</span><span class="mi">31</span><span class="o">-</span><span class="mi">0400</span> <span class="p">[</span><span class="n">scrapy</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">engine</span><span class="p">]</span> <span class="n">DEBUG</span><span class="p">:</span> <span class="n">Crawled</span> <span class="p">(</span><span class="mi">200</span><span class="p">)</span> <span class="o">&lt;</span><span class="n">GET</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">example</span><span class="o">.</span><span class="n">org</span><span class="o">&gt;</span> <span class="p">(</span><span class="n">referer</span><span class="p">:</span> <span class="kc">None</span><span class="p">)</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span> <span class="n">Available</span> <span class="n">Scrapy</span> <span class="n">objects</span><span class="p">:</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">crawler</span>    <span class="o">&lt;</span><span class="n">scrapy</span><span class="o">.</span><span class="n">crawler</span><span class="o">.</span><span class="n">Crawler</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x1e16b50</span><span class="o">&gt;</span>
<span class="o">...</span>

<span class="o">&gt;&gt;&gt;</span> <span class="n">response</span><span class="o">.</span><span class="n">url</span>
<span class="s1">&#39;http://example.org&#39;</span>
</pre></div>
</div>
<p>然后，可以检查提取代码是否正常工作：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//h1[@class=&quot;fn&quot;]&#39;</span><span class="p">)</span>
<span class="go">[]</span>
</pre></div>
</div>
<p>不，它不是。所以您可以在web浏览器中打开响应，看看它是否是您期望的响应：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">view</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
<span class="go">True</span>
</pre></div>
</div>
<p>最后，单击ctrl-d（或在Windows中单击ctrl-z）退出shell并继续爬网：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="o">^</span><span class="n">D</span>
<span class="go">2014-01-23 17:50:03-0400 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET http://example.net&gt; (referer: None)</span>
<span class="gp">...</span>
</pre></div>
</div>
<p>请注意，您不能使用 <code class="docutils literal notranslate"><span class="pre">fetch</span></code> 这里的快捷方式，因为报废的引擎被外壳挡住了。然而，当你离开外壳后，蜘蛛会继续在它停止的地方爬行，如上图所示。</p>
</div>
</div>


           </div>
           
          </div>
          <footer>
  
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
      
        <a href="item-pipeline.html" class="btn btn-neutral float-right" title="项目管道" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
      
      
        <a href="loaders.html" class="btn btn-neutral float-left" title="项目加载器" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
      
    </div>
  

  <hr/>

  <div role="contentinfo">
    <p>
        
        &copy; 版权所有 2008–2020, Scrapy developers
      <span class="lastupdated">
        最后更新于 10月 18, 2020.
      </span>

    </p>
  </div>
    
    
    
    Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a
    
    <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a>
    
    provided by <a href="https://readthedocs.org">Read the Docs</a>. 

</footer>

        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
  
 
<script type="text/javascript">
!function(){var analytics=window.analytics=window.analytics||[];if(!analytics.initialize)if(analytics.invoked)window.console&&console.error&&console.error("Segment snippet included twice.");else{analytics.invoked=!0;analytics.methods=["trackSubmit","trackClick","trackLink","trackForm","pageview","identify","reset","group","track","ready","alias","page","once","off","on"];analytics.factory=function(t){return function(){var e=Array.prototype.slice.call(arguments);e.unshift(t);analytics.push(e);return analytics}};for(var t=0;t<analytics.methods.length;t++){var e=analytics.methods[t];analytics[e]=analytics.factory(e)}analytics.load=function(t){var e=document.createElement("script");e.type="text/javascript";e.async=!0;e.src=("https:"===document.location.protocol?"https://":"http://")+"cdn.segment.com/analytics.js/v1/"+t+"/analytics.min.js";var n=document.getElementsByTagName("script")[0];n.parentNode.insertBefore(e,n)};analytics.SNIPPET_VERSION="3.1.0";
analytics.load("8UDQfnf3cyFSTsM4YANnW5sXmgZVILbA");
analytics.page();
}}();

analytics.ready(function () {
    ga('require', 'linker');
    ga('linker:autoLink', ['scrapinghub.com', 'crawlera.com']);
});
</script>


</body>
</html>